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Introduction 


In most communities the responsibility for crime prevention and the 
apprehension of offenders falls fo the police. Generally, the police 
are considered the community agency with the expertise to deal 
with criminal and other socially disruptive behaviours. It is Only in 
the last twenty years, however, that communities have become 
involved in crime prevention in a co-ordinated way. Realising the 
limitations of leaving the sole responsibility for crime fo police, 
communities have assumed greater responsibility for their own 


safety and security. Many productive partnerships have arisen. 


By becoming involved in crime prevention as auxiliary officers, as 
members of neighbourhood police committees or as participants in 
programs such as Neighbourhood Watch, citizens have begun to 


assume greater responsibility for the well being of their community. 


In many communities, the police are not the principal agent in the 
prevention of crime. Due to geographic isolation, the absence of 
funds for services or the inability of police agencies to meet com- 
munity needs, the community has taken the lead in establishing 
and maintaining crime prevention initiatives. The culture of many 
Communities, such as First Nations, has meant that a philosophy of 
broad community well-being with wide citizen responsibility has 
prevailed for many years. Such a philosophy is a foundation of 


these cultures. 
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Crime prevention is, typically, only a small part of the solution to 

larger and more complex social problems. The approach of many 
First Nations communities is to strategise and plan in an holistic way 
which incorporates strategies such as inter-generational participa- 


tion and broad community responsibility. 


With the changing dynamics of policing, community safety and 
security combined with limited resources, there has been a parallel 
need to review crime prevention practices. This manual has been 
developed to facilitate the evaluation of crime prevention pro- 


grams in First Nations communities. 


With geographic diversity and differing levels of formal policing, a 
broad range of people will be involved in the delivery and evalua- 
tion of these programs. This manual has been developed as a re- 
sult of a contribution agreement with the Ministry of the Solicitor 
General of Canada, Aboriginal Policing Directorate for use by any- 
one in First Nations communities, including police, who may be 


given responsibility for the evaluation of crime prevention initiatives. 


While the manual provides a step by step approach to the evalua- 
tion of crime prevention programs, the manual must be only one of 
the tools used to establish, maintain and review programs. Pro- 
grams, and the manuals designed for their evaluation, will work 
most effectively when the community and community agencies 
are engaged in a co-ordinated approach to community safety 


and security. 
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First, A Few Points To Ponder. The Six W’s 


e Who ls This Manual For? 

e What is Evaluation? 

e What Should be Evaluated? 

e Why Should We Evaluate? 

e When Should We Evaluate? 

e Who Should Do the Evaluation? 


Who Is This Manual For? 


As policing and crime prevention become more community ori- 
ented on the one hand and as the community becomes more in- 
volved in community safety and security on the other, there is in- 
creasing likelinood that non-police personnel will be actively en- 


gaged in the implementation and evaluation of programs. 


This manual is intended as a practical workbook for those who 
have assumed responsibility for the evaluation of a crime preven- 
tion program. The evaluators are likely to be program personnel, 
Band Council staff or community members. The evaluator may 
also, of course, be a member of a police agency. It is assumed 
that the person with the responsibility for the evaluation is not nec- 
essarily well versed in the intricacies of evaluation methodology but 
wishes to assess the success of crime prevention program operat- 


ing within their community. 


The manual is designed for communities with limited access to the 
sophisticated hardware, software and personnel often available to 
communities in urban jurisdictions. Accordingly, the manual does 


not address such things as the use of specialised computer soft- 
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ware or sophisticated statistical analysis. It is assumed that the 
evaluator requires a basic understanding of program review and 
that a more in-depth knowledge can be acquired through spe- 


cialist assistance. 


The manual is user friendly and focuses on applied evaluation. It is 
not a theoretical treatise. The terms used are practical and a glos- 


sary of terms can be found in the appendix. 


What is Evaluation? 


Evaluation is a way of checking to see whether a program is doing 
what it was intended to do. Evaluation is an assessment of whether 
a program is meeting its objectives and, ideally, to what extent and 
why (or why not) the goals have been achieved. How successful 


has the program been? 


Evaluation is merely one stage of the process of addressing a 
community crime problem. Evaluation of a program is an integral 
part of the process of program development and maintenance 
designed to ensure that the community response to a safety or se- 


curity issue works in the most effective and efficient manner. 
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WHERE EVALUATION FITS IN A CRIME PREVENTION PROGRAM 


Ideally, the implementation and evaluation of a program are inte- 
gral parts of a planned process. The principal stages of a crime 


prevention program are: 


e Realisation - the recognition that there is a safety or security problem 
in a community and that some form of action should be taken. 


e Identification - the definition of the problem based upon statistics, in- 
terviews, observation or other methods so that the community is pro- 
vided a focus for action. 


e Action plan - the formulation of a plan which will lay out a series of ac- 
tions to address the issue and assign responsibility for the each of the 
actions to ensure realisation of the project goal. 


e Implementation - Undertaking each of the tasks in the action plan in 
sequence after securing the necessary funding and ensuring sufficient 
resources to sustain the program for the duration of the plan. 


e Monitoring - The gathering of quantitative and qualitative data over 
the duration of the program such that, periodically, an assessment of 
progress can be undertaken. 


e Evaluation - The assessment of information at appropriate points in the 
program to gauge its success in meeting the program goals. As we 
will see later, the review can be a ‘process’ or ‘impact’ evaluation. 
Process evaluation refers to the assessment of each stage of the pro- 
gram to ensure that the action plan is being carried out appropriately. 
The Impact evaluation assesses the degree of change which has oc- 
curred as a result of the program. 


e Feedback - Provides for the adjustment of the program or subsequent 
offshoots of the program to accommodate the findings of the evalua- 
tion. There is little point in introducing the program or undertaking the 
evaluation if the findings are not considered as a basis for further ac- 
tion in the community. 


Evaluation, therefore, is a planned and important part of a crime 
prevention program. The nature and form of the evaluation must 
be anticipated at the program planning stage and should not be 
considered an ‘add-on’ as the term of the program comes to an 
end. However, in reality, many initiatives are introduced with lim- 


ited planning regarding the mechanics of an evaluation and, fre- 
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quently, with little concept of the need for an assessment of the 


effectiveness of the program. 


What Should be Evaluated? 


Crime prevention takes many forms. At one end of the spectrum, 
there are long term programs which are developed to target one 
aspect of complex community problems such as youth suicide, ra- 
cial violence or child abuse. These programs are often multi- 
agency funded and supported by federal or provincial policy or 
legislation. As crime prevention practitioners, you may be respon- 
sible for the implementation of one part of these programs in your 


community. 


At the other end of the scale are the local initiatives which may be 
individual initiatives on the part of police officers to resolve a 


neighbourhood, a family or even a personal issue for a citizen. 


These initiatives can all contribute to the prevention of crime. 
Shouid one evaluate all these initiatives¢e All these crime preven- 
tion efforts take time and, therefore, money. They will all have 
some impact upon the subject and upon the resources of the 
community. They should all, therefore, be subject to some form of 
“assessment” to determine whether the time and effort is worth- 
while. Evaluation projects also take time and cost money. The 
level of sophistication of the evaluation depends upon the amount 


of investment expended on the initiative. 


A full, complex evaluation to determine whether an officer should 


counsel a family on a one time basis is obviously not worthwhile. 
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However, even the most minor of initiatives should be subject to 
some form of assessment even if this is a ‘mental tally” by the offi- 
cer or the supervisor of why this action is being done, the intended 
benefit of the action, how will the officer gauge the success, and 
whether the initiative should be tried on another occasion. This is 
sort of an ‘intuitive’ assessment. As we shall see later, these steps 
are the foundation for an evaluation. These same principles can 


be applied to any crime prevention program. 


So, what should be evaluated? As we have noted, every crime 
prevention initiative should be evaluated. However, it is obvious 
that not every initiative should be subject to a comprehensive 
evaluation. The amount of energy directed to the evaluation 


should, roughly, be in direct proportion to, 


e the scope of the program (how many people are ‘touched’ by the 
program) 


e the breadth of effort of the program (how many people are devoted 
to operating the program) 


e the longevity of the program (if along term investment is anticipated, 
you should assure that effort is not fruitless) 


e the potential community impact of the program (the politics), 
e any contractual requirements to evaluate, and 


e cost (time, dollars and effort) of the evaluation. 


This manual lays out the principles and steps to be taken in the 
evaluation of any crime prevention program. The crime preven- 
tion practitioner needs to gauge the intended, or likely, overall im- 
pact of the program on resources, time, or the community to de- 


termine to what depth the evaluation should be pursued. 
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The content of the manual then needs fo be selectively ‘geared 
back” and the methods and effort in the evaluation tailored to the 
significance of your crime prevention program. Following the prin- 
ciples outlined, considering each step and deciding whether It is 
relevant to your program will ensure that whatever the dimensions 


of the program, you will have subjected it to a valid review. 
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Why Should We Evaluate? 


Crime prevention practitioners and members of community groups 
involved in prevention initiatives frequently feel that a program has 
worked. Intuitively they may know that incidents of crime are re- 
duced or that the events which gave rise to community concern 
have been resolved. However, in times of limited resources 
stakeholders have a right to know whether the initiative actually 
has worked, what impact it had upon the problem which had 
been identified and even if it was the specific program that had 
the impact or whether it was some apparently unrelated action. 


The evaluation also lays the foundation for planning. 


REASONS TO EVALUATE 

Every evaluation is important in that it can help to identify the 
strengths and weaknesses of a program and help the community 
or agency to plan strategies. There are several reasons for demon- 
strating whether and how the initiatives produced the promised 


outcomes: 


e Most initiatives invoive considerable expenditure of money or time. 
Are these resources well spent or should they be diverted to other 
strategies? 


e Many programs are funded by an outside resource, contractual re- 
quirements often require an evaluation to ensure that such funds are 
well spent. 


e Information gleaned from evaluations should be shared with other or- 
ganisations as a basis for (i) the introduction of the program in another 
community, (ii) the decision not to introduce the program in another 
community or (iii) to adjust the structure of the program to meet local 
conditions in other locales. 


e The community needs to know whether its resources are being used in 
the most effective way. 


e The implementation of a new program may create a sense of security 
in the community. Is this assumption is well founded? 
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e People who are involved in programs as victims or offenders need to 
know that the initiative appropriately addresses their needs. 


e Toidentify or justify the next courses of action in an on-going program. 


e There may be many reasons for the success or failure of a program, 
such as inappropriate training, failure to support the desired behav- 
iours or contradictory policies. These factors should be identified to 
ensure that the correct conclusions are drawn and that subsequent 
actions, such as renewed funding, are directed to the appropriate ac- 
tions. 


e Tomotivate staff by highlighting successes. 


When Should We Evaluate? 


We have already noted that evaluation is an integral part of any 
program. The planning for an evaluation should be undertaken 
during the planning stage of the crime prevention program. 
Evaluation can be time consuming and expensive. If an evaluation 
does not address the relevant factors then the effort is futile. Like- 
wise, if the program is not evaluated then the time and effort in- 


vested in the initiative also may be wasted. 


The planning of the evaluation at the outset of the initiative is par- 
ticularly important because the evaluation may require certain in- 
formation which must be built into program planning. For example, 
data (or information) might have to be gathered through the use 
of a special form or, to permit an effective evaluation, minor ad- 
justments may have to be made to the operational practices or 
program design. For example, for comparison purposes, different 
approaches to the crime problem may have to be taken in within 
the community or in two different communities or we may nave to 


ask the members of a Victim Services Unit to gather information on 
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the presence of alcohol at domestic incidents or when children are 


taken into custody. 


Where a process evaluation is undertaken, each stage of the pro- 
gram must be assessed as it is undertaken to ensure that the pro- 
gram is introduced in the correct way according to the project 
plan. Therefore, a process evaluation must ideally be ready to in- 


troduce as soon as the initiative begins. 


The value of a process evaluation is that it allows the evaluator (or 
the program co-ordinator) to make adjustments to the program to 
ensure that it is ‘on course’. For example, staff may have to re- 
ceive additional training or the types of incidents attended by a 
Victim Services Unit may have to be expended or reduced. The 
adjustments which are made however, should not alter the pro- 
gram process to such an extent that the original objectives are no 


longer valid. 


An impact evaluation assesses whether the program achieved its 
desired objectives. By definition, it must therefore be conducted 
sometime after the program has, in a substantive way, been com- 
pleted. Sufficient, planned time must have elapsed to allow the 
program adequate time to take effect. The exact timing of the 


impact evaluation is dependent upon the nature of the program. 
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The Impact evaluation, therefore, presents a few challenges. The 
methodology must be anticipated based upon a clear under- 
standing of the objectives of the program and the anticipated 
outcomes. In addition, the evaluator must assess the potential 
problems arising from the implementation and the conduct of the 


program. 


Nevertheless, the Impact evaluation is crucial in determining the 
success of the crime prevention program. We will examine these 


issues later. 


Who Should Do the Evaluation? 


This is intended to be a self evaluation manual. It implies, therefore, 
that the evaluation is undertaken, or co-ordinated, by a 


stakeholder from within the community. 


The evaluator could, therefore, be, 


e individuals working within the program, 
e aperson within the agency who does not work in the program, 
e acitizen from the community, or 


e someone from another nearby community 


Alternatively, the evaluation can be taken with assistance from 
someone from outside the immediate community but the project 


should be co-ordinated by a community representative or group. 


Evaluation methodologies have become very complex. Sophisti- 


cated statistical analysis is a requirement for publication in many 
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academic publications. Few organisations, however, have access 
to either the personnel or the software or hardware to undertake 


an evaluation of this level. 


On occasion, the staff or students from a local college or university 
may be recruited to undertake an evaluation as part of a course. 
Involvement in community programs provides academic institutions 


with the opportunity to give students access to practical fieldwork. 


Likewise, external consultants may be engaged to evaluate pro- 

grams. Such support is likely to cost money in the form of consulting 
fees and, in remote communities, would also incur significant travel 
costs. If external experts are used as evaluators, project managers 
should ensure that knowledge transfer is assured to equip local staff 


or residents with the skills to undertake future evaluation projects. 


Employees of community agencies, such as police officers, may be 
part of a larger organisation which can provide research support. 
This is particularly likely if the community is selected as the site for a 


pilot program which may be considered for wider use. 


Peer evaluation allows for persons from other similar agencies or 
communities to undertake the evaluation. The peer evaluation 
may be undertaken on a reciprocal basis with a member of your 
community later participating in the evaluation of the programs of 


the other agency or community. 


In most situations, however, it will fall to agency workers or com- 
munity members to plan, develop, implement and evaluate a 


crime prevention program. As distance education becomes more 
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common, many citizens and agency personnel are now well 


versed in program development and evaluation. 


Whoever is ultimately responsible for the evaluation, the involve- 
ment of several people as a ‘sounding board’ for ideas and to pro- 
vide input to the evaluation strategies will ensure a more compre- 
hensive product. The inclusion of Elders, women, youth or Band 
Council members on a steering committee will ensure a more 
comprehensive perspective on the evaluation process and pro- 


gram. 


The involvement of a co-ordinating group will also permit the shar- 
ing of responsibilities for various aspects of the evaluation. Overall 
responsibility must remain with one person who will orchestrate the 


activities. 


QUALIFICATIONS OF EVALUATOR 

Although, ideally, a member of the community should Undertake 
the evaluation, there are several criteria to consider when select- 
ing a person to undertake the planning and evaluation of a crime 


prevention program: 


e The person(s) should be as objective as possible. The person should 
not be directly involved in the program as a victim or offender. 


e The person should not have strong vested interest in the outcome of 
the program. For example, the evaluators continued employment 
should not be dependent upon the reported success and continua- 
tion of the program. 


e Where confidentiality of data is important, as far as possible, the per- 
son should have nothing in his/her background which may appear to 
jeopardise the integrity of the data. 
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e Although difficult in smaller communities, the evaluator should not 
have friends or relatives who are involved in the program as victim, 
offender or agency worker. The perceived confidentiality of data 
may be compromised and some program participants may be reluc- 
tant to share their opinions. 


e The person should be seen to be objective rather than have a strong 
view on the important pro's and con's of the program. 


e Where possible, the person should have some understanding of pro- 
gram planning, evaluation and statistics. Hopefully, this manual will 
provide some background in these activities 


The peer evaluator must be, 
e capable of undertaking a valid review, 


e prepared to keep confidential information obtained in the evaluation, 
and 


e prepared to be open and objective regarding the findings and willing 
to share the results with the community. 


However, given factors such as geographic isolation and funding 
limitations, it is more important that the crime prevention programs 
are initiated and subject to some form of evaluation, rather than 
preventive programs not being introduced because of limited op- 
portunity for a sophisticated assessment of effectiveness and effi- 


ciency. 


Perivale + taylor 


First Nations Crime Prevention — Self Evaluation Manual 16 


Do We Have The Resources To Undertake The Evaluation? 


If funding has been secured from an outside agency, there will fre- 
quently be a requirement that an evaluation component is built 
into the program. Consequently, sufficient funds should be avail- 
able for an evaluation if the program planners have developed the 


program budget appropriately. 


Occasionally, evaluation may be an after thought and funding must 
be sought after the program has been introduced. Post- 
implementation planning of evaluations may also arise where it is 
realised, after the fact, that a series of actions constitute a program 


and have led to an outcome which should be assessed. 


Federal and provincial government agencies are occasionally willing to 
fund the set-up and evaluation of innovative programs which can 
be transferred to other communities. Private companies are usually 
more willing to sponsor capital projects such as buildings and, 
sometimes, high profile crime prevention initiatives rather than op- 


erational costs such as crime prevention evaluation. 


The manual provides a structure for developing a proposal for the 
funding of an evaluation project. The possible costs to consider in 


developing a budget are listed in the appendices. 


Frequently, it is possible to undertake an evaluation with limited re- 
sources. Basic evaluations are often sufficient to determine the 


relative success of many programs and cost may be limited to the 
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evaluators’ time and incidentals such as the printing of forms for 
data gathering and the purchase of stamps if mail surveys are con- 
sidered. Personal computers now have the capacity to provide 
relatively sophisticated support for calculations, tables and report 


preparation. 


In addition, the joint participation in an evaluation with a nearby 
community which has introduced a similar crime prevention pro- 
gram may help to defray costs. As noted below, it is therefore im- 
portant to secure an understanding of other crime prevention pro- 


grams and evaluations which have been implemented. 


Of course, whatever funding you may have acquired should be 
allotted croportionately to the different stages of the evaluation. If 
you choose to undertake a method of information gathering such 
as a community survey, this will require more funding than, say, a 


series of interviews with agency staff. 
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How To Use The Manual. 


The manual is a ‘how-to’ guide for use by agency personnel who can 
work through the document, step by step, to assess whether the 
crime prevention program has achieved its objectives. The manual 
can also be used as a structure to develop a proposal to seek 


funding for an evaluation. 


You will note that detail is provided by supplying information, such 
as, the pro’s and con's of different methods to gather information. 
It is presumed that, on many occasions, you will be Undertaking an 
evaluation with limited outside support. The detail will equip you 
with the technical knowledge that you need to consider the spe- 
cifics of the particular program which you are evaluating and the 
characteristics of your community and to develop a work plan 


which is tailored to your needs by using the most appropriate tools. 


Ideally the evaluation plan will be built into the program from the 
outset. In reality, however, the evaluation of a program is often 
introduced after the fact. The manual attempts to accommodate 
the need to sometimes catch-up and to fill-in the gaps after the 


program has started. 


The manual lays out the principles and steps to be taken in the 
evaluation of any crime prevention program. Not all programs will 
need this amount of detail. Not all programs will have all the in- 
formation noted available to undertake an evaluation. However, 
this manual will equip you with the skills to tackle almost any 


evaluation project. 


Perivale + taylor 


First Nations Crime Prevention — Self Evaluation Manual 19 


The crime prevention practitioner must gauge the intended or likely 
overall impact of the program on resources, time, or the commv- 


nity to determine to what depth the evaluation should be pursued. 


The content of the manual then needs to be selectively ‘geared 
back" to suit the scope and complexity of the program. Following 
the principles outlined, considering each step and deciding 
whether it is relevant to your program will ensure that whatever the 


size of the program you will have subjected it to a valid review. 


The manual refers to the evaluation of programs. It does not address 
the design or execution of programs. These are specialised topics in 
themselves and the reader is directed to other materials for informa- 


tion regarding these topics. 


WORKING CASE STUDY 


ee 1 eee ee ee eee 
Per 
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POINTS TO KEEP IN MIND 


At specific points in the manual there are a series of questions 
which you can pose to yourself fo assess where you are in the pro- 


cess. These questions will help you focus on the principle decisions. 


Ask yourself: 
What is the purpose in undertaking this evaluation? 


Given the progress of the program, is it the appropriate time 
to undertake an evaluation? 


Do I have the resources as far as skills and funding? 


If I am doing the evaluation alone, can I organise an advisory 
committee from the community? 


All this might sound overwhelming. Don’t worry. Things will be- 
come more clear as you read on and begin fo look at the program 


you are going to evaluate in your community. 
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Getting Ready. 
Undertaking the Background Research 


The program to be evaluated must be seen in its operational and 
theoretical context. What is this program meant to be achieving? 
The background documents and literature Upon which the pro- 
gram was based should be reviewed. Where did the idea for this 
program come from and why? If possible, conduct a literature 
search or make a few telephone calls to determine whether other 
programs of a similar nature have been introduced and whether 


these programs have been subject to an evaluation. 


Local agency contacts such as (other) police officers or govern- 
ment department personnel may provide information. The Internet 
and distance education facilities can give access to a wide variety 
of national and international literature, and follow-up via telephone 
may provide hard-copy or e-mail versions of the background ma- 


terials. 


If the program has generated local interest then local media files 


may provide background information. 


Consult the stakeholders in the program. In your discussion, focus 
on what the program is and was intended to be rather than on 
what form it now takes. Understanding what is currently happening 


will come later. 


FACTORS TO CONSIDER 
It is important to view the evaluation (and the original program) in 


its broad context so that, as an evaluator, you are able to later as- 
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sess issues Which have to be considered in the evaluation. Con- 


sider factors such as the, 


Community - Geographic size, population and age breakdown, 
general crime trends, suicide rates, divorce figures, cultural and 


historical backgrounds and migration factors. 


Economic factors - Band and individual income and sources, em- 
ployment and unemployment, opportunities for off-reserve income, 
businesses which generate income that remains in the community 


rather than businesses where the money moves out. 


Socio-political- The views of the Band Council, the views of the El- 
ders, influence of the provincial and federal governments, impact 
of other factors such as the media and major income generators 


such as casino’s or oil companies. 


Sources such as census data, government inquiries, newspaper files 


and oral traditions may provide background information. 


SUMMARISE FINDINGS 
When the information has been gathered, 


e Prepare a brief description of the operation of the program. Explain 
who, what, when, where, is (or was) the intended focus of the pro- 
gram and how it was supposed to operate. 


e Define the purpose of the program. What were the program objec- 
tives? What was the original vision for the program when it was intro- 
duced? What specific community order or crime problem was the 
program intended to address? 


e Describe similar programs in other communities. How have these pro- 
grams been operated? Were the programs evaluated? How were 
they evaluated? Were they successful? 
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Don't forget, at this time, there is no need to concern yourself with 
whether, or even how, the program currently works. We are only 
looking at what it was meant to be doing and in what context it is 


operating. 


Ask yourself: 


. Have I fully described this program as it is intended to oper- 
ate in this community? 


Have I defined in a concise way the objective(s) or goal(s) of 
the program? 


Have I described the important community factors which may 
influence the operation of this program? 


Do I understand if, and how, the program operates in other 
communities? 
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Setting Evaluation Objectives: 


The objectives provide a focus for the work. What were the original 
objectives of the crime prevention program? Understanding these 


will assist you in determining the objectives of the evaluation. 


Then, following on from this, what are you trying to determine in 


your evaluation? 


An objective states clearly what you want the evaluation project to 
do. An objective (or objectives, there may be more than one}, is a 
clear statement of the results which you want to achieve. A pro- 
gram and an evaluation plan must be realistic. Select a manage- 
able number of objectives. Perhaps, three fo six is manageable. 
More objectives than this and the evaluation will lose focus. (More 
program objectives than this and the original program will also lose 


focus). 


If there is no clear statement of objectives for the program, and 
you are unable to discover one in your initial talks with stakeholders, 
then you must ‘blue-sky’ to develop possible impacts of the pro- 


gram to provide some focus for your evaluation. 


WHAT IS A STATEMENT OF OBJECTIVES 


e It must be clear so that all participants in the program or in the evalua- 
tion fully understand what the program is intended to do. 


e The objectives must be acceptable to all participants. The goal must 
not only address the problem which has been identified but also be 
ethically acceptable to everyone. For example, nobody should have 
vital services withdrawn as an experimental part of a program or 
evaluation. 


e The objectives must be achievable. Be creative and optimistic but be 
realistic. 
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e The objectives must be measurable so that the evaluator can collect 
information to assess the success of the program, whether it has 
“measured up" to expectations. Later we will describe the types of 
data which will become the foundation for the evaluation. 


e The objectives must be long range. This provides a focus for program 
activities and permits sufficient time to have elapsed for the program 
to have some effect. At the same time, the duration must not be too 
long. Resources are being expended and may be more efficiently 
spent in other programs. 


e The duration of the program must be stipulated. A point(s) in time for 
evaluation must be decided at the outset and a point in the future 
when you will make decisions regarding the continuation of the pro- 
gram should be decided in the planning stage. This will provide focus 
for the participants, ensure stakeholders that an assessment will be 
made at some time and prevent the program simply limping on with 
no evaluation. 


e The objectives should be flexible. As you monitor the program (see 
above) it may be necessary to adjust the program due to unforeseen 
circumstances. It is pointless to doggedly pursue the original goals if 
the circumstances surrounding the crime prevention program have 
changed. There is nothing more futile than doing well something that 
doesn't need to be done at all! 


e Anintegral part of the objectives is assigning responsibility for action. 
Participants must know what role they will play in achieving the pro- 
gram or group goal. 
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INDICATORS: 


If the objective shows what you want the program to do, then we 
must also develop an understanding of what types of information 
you will need to have to know whether this objective has been 
met. The information which we use as a basis for this decision is an 


indicator. 


An indicator is a bit of information (such as an event or a statistic) 
to measure an effect of a program. The information ‘indicates’ 


whether the program has had an impact. 


TYPES OF INDICATORS 


Quantitative indicators - These are essentially numbers or quantities. 


AS We will see below, these are based on data such as the callls at- 
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tended by the police, charges laid or the numbers of children 


taken into care or reports of vandalism. 


Qualitative indicators - These are based upon the ‘quality’ of the 
program activities and may be opinions of participants or agency 
workers. These tend to be descriptive but when collected or ag- 
gregated may show, for example, the percentage of those sur- 


veyed who believe that a program works. 


Short term indicators - These are indicators which are measured 
over a brief interval and although short term indicators are impor- 
tant and tell us something about progress they are likely to be in- 


conclusive regarding the overall success of a program. 


Long term Indicators - These are indicators which are measured 
over the longer term and tend to address the long term impact of 


a program, often over its full duration. 


Perivale + taylor 


First Nations Crime Prevention — Self Evaluation Manual 28 


Ask yourself: 


Have I described the objectives of the program or, if these 
were not specified in the original program outline, have I de- 


veloped objectives for the evaluation? 


Have I determined what types of information will be used to 
indicate whether, and how, the program has worked? 


Deciding the Form of the Evaluation Program 


There are three types of evaluation processes that you can use. 


We mentioned these earlier. Here we will explain a little more. 


Needs Assessment- The needs assessment typically occurs at the 
early stage of the program development process. It will help you 
to siff through the various crime problems which your community 
may face and to focus upon those of the highest priority. Essen- 


tially, what are the needs of the community? 


The needs assessment should not really be part of an evaluation 
program. If should have been the basis for the development of the 
original program designed to address the identified need. We 
mention it here because if a needs assessment has been con- 
ducted, it will provide some useful information to you as a basis for 


your work. 


Process Evaluation (The means to the end) - The process evaluation 
allows the evaluator to monitor the implementation and operation of 
the crime prevention program to assess the effectiveness of its opera- 


tion. 
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The process evaluation may review each or selected components of 
the program, such as participant selection, training of counsellors, the 
speed of the police attending calls, to determine whether each is 
operated appropriately having consideration of the program objec- 
tives. 


Impact evaluation (Whether the end was achieved) - The Impact 
evaluation is usually undertaken at the termination of the program to 
determine the effect of the program. As the name implies, the 
evaluation is conducted to ascertain the impact of the program on 
the original state of affairs. Did the program achieve its objectives? 
The Impact evaluation is the basis for the crucial question regarding 
whether the resources were well spent, did the community ultimately 


benefit and whether the program should be continued. 


There may be more than one point at which the impact evaluation is 
undertaken. For example, it may be appropriate when assessing the 
long term effect of a crime prevention program to evaluate its impact 


at the termination of the program and then again one year later. 


(You may also consider assessing the impact of one portion of the 
program. For example, you could develop a brief questionnaire to 
assess whether the attendees at a meeting found the session to be 


| 


useful.) 
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Ask yourself: 


Given the nature of the program and the stage at which I am 
planning the evaluation, which approach or approaches will best 
allow for an examination of the effects of the program? 


Types Of Information Which Can Be Used: 


There are two types of data which can form the foundation for 


your evaluation. 


Quantitative data - This looks at the quantities in the evaluation. It 
is the counting of results and consists of the statistics and numbers 
which are obtained through the use of measurement tools. This is 
sometimes called ‘hard data’. It is the data which, as we have 


seen above, supports quantitative indicators. 
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Qualitative Data - This looks at the quality of the crime prevention 
program and is based upon the views and opinions of participants, 
stakeholders and others associated with the program. The qualita- 
tive approach is more descriptive. It is sometimes called ‘soft 
data’. It is the data which, as we have seen above, supports 


qualitative indicators. 


The two approaches are not mutually exclusive. An evaluator 
does not have to choose one or the other in an evaluation. In fact, 
some of the best evaluations use a combination of the two ap- 
proaches. Conclusions supported by both quantitative and quali- 
tative data are often stronger than those based on one or the 
other. Also, much of the quantitative data gathered through in- 


terviews is expressed in terms of the quantification of the findings. 
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The quantitative and qualitative approaches require different tools 


for gathering data and different skill in the evaluators. 


The quantitative approach is numbers based and is more definitive 
in the sense that percentages can be compared across programs 
or communities or regarding the situation before the program and 


after the program. (This is called a pre- and post fest.) 


The qualitative approach is a more interactive approach in which 
opinions are asked or documents examined or observations made 
on activities within the program. The qualitative approach is a less 
rigid evaluation design. Although the results of the qualitative ap- 
proach may ultimately be quantified as a statistical result the data 


is based upon opinion and feelings. 
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Deciding the Type of Information Upon Which Your Evaluation 
Will Be Based 


There are three types of data which can be used to determine the 


effectiveness of your program: 


Baseline data - This refers to the known data or information which 
was available before the program began. This provides a basis for 
comparison as it describes the state of affairs before any crime 


prevention action was taken. 


However, baseline data is not just the information which was col- 
lected in the normal course of operating an agency or a program. 
Baseline data may be specifically developed for the purposes of 


the evaluation to provide a basis for a ‘before’ and ‘after’ test. 


Progress data - This refers to the information which you are collect- 
ing as the program proceeds according to the action plan. Like 
baseline data, these data may be the information which is nor- 
mally collected in the course of operations or the you may have to 
develop the data gathering tools for the purposes of getting infor- 


mation for the evaluation. 
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Culminating data - This is the information which is collected at the 
end of the program evaluation term and comprises the closing 
data. As with baseline data, the information may simply be that 
which is collected in the normal function of the agency or pro- 
gram. Alternatively, it may be collected and analysed specifically 


for the evaluation. 
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Ask yourself: 


Considering the nature of this crime prevention program and 
the sorts of information which might be available, which ap- 
proach or combination of approaches will provide the best 
foundation for an evaluation? 


What sorts of information are likely to be available in this 
community? 


Given the type of people in my community and/or the staff 
working in the program, how am I likely to get the most valu- 
able information? 


Where Do We Find The Information That We Need? 


There are several sources for the information which you can use as 


a basis for your evaluation. 


STATISTICS RESEARCH 


Statistics are merely the numbers you can use as a basis for your 
evaluation. This is the hard data which we mentioned earlier. 
Various forms of statistics are compiled by most government, 
commercial and private organisations and agencies. In the normal 
course of business, these operations need such data to monitor or- 
ganisational and individual performance or to determine such 


things as unit cost. 
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The federal government requires that agencies involved in the aa- 
ministration of criminal justice such as police, courts and probation, 
collect, compile and forward to Statistics Canada certain informa- 
tion regarding the operation of the agency. In the case of the po- 
lice, which will deal with the majority of crime prevention initiatives 
in the community, the Uniform Crime Reporting System (UCRS) spe- 
cifically requires that detailed information concerning reported 
crime, arrests, charges and other data is compiled and submitted 
to the federal government for analysis. These aggregate data are 
readily available from the police or from Centre for Criminal Justice 


Statistics of Statistics Canada. 


There are other centrally co-ordinated statistical collections which 
are not a function of Statistics Canada. One such data base is Vi- 
class which is dedicated to the development of an information 
data base concerning violent offenders. This is a highly specialised 
data base and, generally, only accessible to law enforcement 


agency staff. 


As initiatives such as Problem Oriented Policing become more 
common, general statistical information regarding crime and dis- 


order in the community will become more accessible. 


Other statistical data may have to be collected specifically for the 
evaluation program. The type of statistics you may have to de- 
velop will depend upon the specific issue being evaluated and the 


characteristics of the community in which you are working. 
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Owing to the wide variety and volume of data collected, it is im- 
portant that you are specific regarding exactly what information is 
required to facilitate the evaluation. Sometimes it may be neces- 
sary to go beyond the data collected for the purposes of federal 
government analysis and use information collected in case files or 
through a Computer Aided Dispatch (CAD) system which tracks 
information regarding the dispatch of police to incidents and the 


reporting of criminal incidents. 


The policing standards in most provinces now require police agen- 
cies to maintain a statistical data base and a formal case man- 
agement system. Providing confidentiality requirements are met, 
access to this data will provide background to process and impact 


evaluations. 


Annual reports prepared by criminal justice agencies are a ready 
source of information. Periodic reports from the federal government 


will allow cross-cOommunity Comparisons. 


PROBLEMS WITH STATS. 


Wherever criminal statistics are used in an evaluation there must be 
a strong caution. Criminal statistics are more the record of police 
actions than a record of criminal events which occur in the com- 


munity. There are several filters which will influence the accuracy 


of such data. 
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e Not every incident is witnessed or witnessed by someone who cares 
enough to involve the authorities 


e The so-called ‘Victim-less” crimes are rarely reported as the parties in- 
volved have an interest in not revealing the commission of the of- 
fence. 


e Often victims do not know a crime has occurred or simply do not 
bother to report or may be too intimidated to report. 


e Police will not attend every call for assistance. The police often have 
other priorities and, particularly, in remote areas the police are not 
available to deal with many incidents which may receive attention if 
they occurred in an urban area. 


e Even if the police attend an incident, they will not always arrest an 
offender even where sufficient evidence exists fo justify arrest. Many 
factors beyond simply the facts of the case are considered by an offi- 
cer before he/she decides to take a person into custody. Frequently, 
police will not even complete a report of an incident. 


e Frequently enforcement policies influence arrest rates. For example, 
where a provincial policy addresses specifically “Assault of Women” 
and requires the police to arrest an alleged offender, the police are 
more inclined to arrest than treat the incident in some other way. They 
are also more likely to arrest a male offender in such a circumstance 
than a woman who has assaulted her male partner. Also, some police 
agencies encourage the use of counselling by officers as a response 
to limited court facilities. 


e Where police statistics are biased, the same bias will be reflected 
throughout the criminal justice system. 


e The culture ina community often means that incidents are dealt with 
in ways that do not lead to official reporting of an incident. 
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USING THE STATS. 


e Publications available from UCRS will provide you with an under- 
standing of the breadth of statistical information which is available 
concerning the activities of the agencies involved in the administra- 
tion of criminal justice. 


e Decide which group of statistics is relevant to the focus of the evalua- 
tion. These data may be available in the public reports issued by Sta- 
tistics Canada, other data can be ordered through Statistics Canada 
for a cost. The exact cost depends upon the complexity of the com- 
puter run required to generate the data you require. 


e Review the annual reports of the police or other agencies which may 
have an impact on the operation of the subject program. These re- 
ports will provide a more local flavour and they are likely to be more 
detailed than you would find in federal government data banks. 


e Remember the caution when analysing police and other agency sta- 
tistics Concerning crime. The data is more an account of agency ac- 
tivity than a true reflection of the crime and disorder which may exist 
in the community. 


e Aggregated data from local! police files will show significant detail re- 
garding the geographic distribution of incidents reported and calls at- 
tended, the distribution of workload throughout the 24 hours and 
through the week. Use this data for comparative pre-and post- 
assessment of program impact and for cross-jurisdictional comparn- 
sons. 


e As you review the statistics to be used in the evaluation consider the 
factors which may influence the accuracy of the data. Allow for this in 
deciding your method of collection and other information sources 
which you may use. 


e Look for ‘groupings’ of statistics, such as changes in the rates of van- 
dalism throughout the year, which will allow for the measurement of 
change through increases or decreases over the duration of the pro- 
gram. 


e Insome communities there may be few stats kept on crime or disorder 
problems. It may be necessary for you to develop a process to collect 
this information over the course of your evaluation. 
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Ask yourself: 


Given the objectives of this program what sort of stats infor- 
mation do I need? 


Do any of the community agencies, such as police or the Band 


Council, compile stats. which may be useful in this evaluation? 


If the stats which are kept are not entirely suitable, is it 
possible to develop a process to collect stats. information for 
the purpose of this evaluation? 


FILE REVIEWS 


The examination of files and unpublished reports relevant to the 
program being evaluated provide valuable background regarding 
the development and operation of the program. Examined in the 
preparatory stages of the evaluation, the review can place the 


program in Its issue context. 


In larger urban centres, file information is offen maintained in 
Management Information Systems (MIS). In smaller centres, it is less 
likely that such systems, with the relatively easier methods of ex- 


tracting data, are available. 


Whether an automated or manual system is used, the evaluator first 
needs to know what information is available. The agency man- 
ager (or information systems manager in larger organisations) 
should be able to explain the range of data and what might be 
applicable to the particular evaluation project. Remember the 


information on file is not an evaluation in itself. The data which is 
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collected is only, collectively, bits of information which can be used 


in an evaluation. 


The evaluator should not only draw client or victim or incident data 
from files. In gaining an understanding of the process, information 
such as position descriptions, training schedules, business plans and 
budget information will provide background regarding the effec- 
tiveness of the support elements of an organisation. For example, 
a crime prevention program may be designed and implemented 
such that the opportunities for success is maximised, however, if 
personnel are not trained to operate in the different work environ- 


ment then the program may falter. 


The evaluator should formulate some idea of the information which 
is required, based upon indicators, before approaching the file or 
information bank holder. General ‘fishing expeditions’ will not en- 


courage a sense of confidence in the evaluation or the evaluator. 


Much of the data which is collected as part of the routine man- 
agement and operation of an agency is collected for local pur- 
poses and may not be submitted for federal compilation. Informa- 
tion such as addresses of offenders or victims, times of calls at- 


tended by police, daily records of school attendance are often re- 
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tained at the local level. Access to such information may require a 


confidentiality agreement. 


DOING A FILE OR DOCUMENT REVIEW: 


e The file search and review should be one of the first steps in gathering 
background data. The method is relatively non-intrusive, is low cost 
and is often high yield. 


e Discussion with stakeholders and program staff will provide information 
concerning files and documents which may be relevant to the 
evaluation. Reports such as needs assessments which may have 
been prepared at the outset of the program may reveal community 
profiles, the communities perception of crime issues which should be 
addressed and the rationale for such decisions. 


e As files and documents are reviewed, gather information which pro- 
vides you with a broad understanding of the developmental context 
and operations of the program which is the subject of the evaluation. 


e Take notes on the issues and note names of those involved in the pro- 
gram (or their agency positions). 


e Maintain a list, perhaps on file cards of documents used in the review. 
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Ask yourself: 


Given the objectives of this program what sort of file infor- 
mation do I need? 


Do any of the community agencies, such as police or the Band 
Council, keep file information which may be useful in this 
evaluation? 


If the files which are kept are not entirely suitable, are the 
required files available in any other location? 


Do I need any sort of security clearance to have access to 
this material? 


SURVEYS 


Surveys can be distributed to large numbers of people with the 
benefit that a broad spectrum of views is facilitated and more 
people are provided the opportunity to offer an opinion on the 


program. 


Surveys can be conducted prior to and at the end of a program to 


assess, for example, changes in attitudes or perceptions. 


The principle survey methods are telephone and mail-out (includ- 


ing general distribution to employees of an agency). 


Telephone surveys - Provide the opportunity for large sampling of 
views from the relative ease at the end of a telephone. However, 
many isolated communities may have restricted access to tele- 
phones. In addition, given wide geographical areas this method 
can be costly. Depending upon the size of the survey, significant 
cost can be incurred if a private company is used to conduct the 


work. The advantage is, however, that the staff are trained to 
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conduct this type of work, call-backs are possible at a range of 
times without too much inconvenience to the evaluator and, by 


arrangement, responses are received tabulated. 


Unfortunately, response rates are fairly low. This depends upon the 
issue covered however. Having dinner disturbed by a telephone 
survey on crime and policing issues may be more acceptable than 
less ‘interesting’ topics that are unlikely to directly affect the re- 


spondent. 


Mail out surveys - Mail-out surveys suffer from similar problem as 
telephone surveys. A potential respondent is less likely to respond 
to a survey which arrives through a mail box than through a tele- 
phone contact. There is often a need for follow-up to encourage 
to potential respondents and even then response rates are gener- 


ally low. 


There is greater chance of a higher response rate if questionnaires 
are distributed to, for example, employees of an criminal justice 
agency as the issues are likely to be more relevant and follow-up is 
easier. However, unless questionnaires are confidential, re- 
sponses may be tailored in anticipation of the answers being seen 


by other agency personnel. 


In all survey preparation, specific questions and the format of ques- 
tionnaires must be prepared very carefully to elicit resoonse which 
are relevant without introducing bias in the respondents answers. 

Questions cannot ‘lead’ the respondent but must give a clear indi- 


cation of the types of information which is being sought. 
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Survey questions can be either open-ended, multiple choice or 


closed. 


An Open ended question invites a narrative opinion from the re- 
spondent. Although this gives the respondent more latitude to an- 
swer in exactly the way that he or she wishes, the open-ended 
auestion is difficult to analyse. The evaluator must individually read 
and assess the answer to deduce the specific point that the re- 


spondent is trying to make in the response. 


The multiple choice question requires that the evaluator anticipate 
the range of possible responses to a question. This range is then en- 
tered on the answer sheet for consideration by the respondent 
who marks the appropriate response or responses which he or she 
consider most accurately reflects their view. It must be made 
clear whether the respondent has an option of one or more re- 
sponses to each question and a category such as ‘Not known’ or 


‘Other’ (with space for elaboration) should always be permitted. 


Included in the multiple choice form of questionnaire is the ‘scale’ 
response in which respondents are invited to provide a view on a 
graded scale of, say, 1 to 5, with the numbers corresponding to 
progressive levels of preference. Alternatively, the numbers may 
refer to a series of non-progressive options such as the respondents 


view on the important crime issues in the community. 
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A field test, by asking a few respondents the draft questions, prior to 
the beginning of the evaluation will frequently allow for refinement 


of the categories in a multiple choice questionnaire. 


Closed answer format gives the respondent only the opportunity to 
answer with a restricted set of options, such as, ‘Yes’, ‘No’ or ‘Not 


sure’. 


Accurate sampling is crucial to a valid survey. The potential recipi- 
ents of the survey must accurately represent the breakdown in the 
population of the community or the organisation which is being 
surveyed. The evaluator must, therefore, know precisely what is the 
make-up of the total group of victims or offenders or community 


citizens or employees before accurately determining the range of, 
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and number of, people who will be surveyed. Statistics Canada 
census data will provide demographic background on very spe- 


cific geographical areas. 


All questionnaires should be field tested prior to the full survey. For 
example, pre-testing through role playing with agency staff may 
reduce the error rate. The development of good surveys are quite 
difficult and time consuming. Through your background research 
you can often find examples of surveys conducted in other com- 


munities. 


THE PROS AND THE CONS OF SURVEYS 
The Pros: 


e Information is aggregated or grouped for analysis, so 
no-one is identified individually. The anonymity may 
encourage more people to participate. 


e Conducted using a large number of respondents, gen- 
eral views on issues are captured rather than the opin- 
ion of a few individuals which may reflect a bias. 


e Easy to administer and provides input to the evaluation 
by large groups of people. 


e Closed questions allow for easy tabulation of responses 


The Cons: 


e Telephone or mail-out surveys are expensive and gen- 
erally return or answer rates are low. 
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e They are difficult to develop as the questions must be 
exact to focus specifically upon the issue(s) which you 
want to examine. 


e Inmany communities, telephones may not be avail- 
able. 


e Inmany communities, telephone or writing on a survey 
may not be the preferred form of communication. 


e The sample, or people who are asked to respond, must 
be representative, that is, the people who are given an 
opportunity to respond must reflect the total popula- 
tion, such as, potential users of the program or tax 
payers or police officers. The sample should reflect only 
the bias which naturally appears in the target group, 
such as males and females, age, income levels, 
agency ranks, location of residence or victims versus 
non-victims. A random sample, that is distributing the 
survey over a wide population in the hope of including 
all these biases, is an effective way of gathering infor- 
mation. 


DEVELOPING A SURVEY 


e Decide what information you want to know, what are the issues to be 
addressed bearing in mind the evaluation objective. 


e Determine the sample group who will receive the survey. This will de- 
termine the nature and content of questions. You may need to col- 
lect basic demographic data to ensure that your sample is represen- 
tative. Question wording should allow for educational, language and 
cultural differences. 


e Determine a list of topics which can become the focus of your ques- 
tions. 


e For each topic, develop a series of questions which focus on the nub 
of the topic. Be aware that too many questions will put people off, 
not enough will not provide the information that you are seeking. 


e Decide what form of question is most appropriate for each topic. |s it 
appropriate to ask a question which can be answered by ‘yes’ or ‘no’ 
or is a more detailed answer which allows the respondent to explain a 
feeling or a perception more appropriate? 
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e If you decide upon a question which allows selection from a multiple 
choice, have you considered all possible responses and then reduced 
these to a reasonable number of options? Somewhere between five 
and ten is a reasonable number. Allow for ‘Don't knows’ and ‘Not ap- 
plicable’ 


e Review your questions. 


e Is each question clear regarding what type of information you 
are trying to receive? 


e Will the wording of the question influence the nature or tone of 
the response? 


e Does each question pose just one question and not more? 
e Test the questions on a colleague. 

e Does he or she understand them? 

e Are there any that are repetitive? 


e Are there other responses in the multiple choice which you had 
not anticipated? 


e How much time does it take to complete the survey? 


e Place the questions in a logical order. If a topic follows on from, or de- 
velops on, another question make these consecutive. Number the 
questions. 


e Prepare an introduction which briefly explains the purpose of the sur- 
vey and the goal of ensuring confidentiality of information. This will be 
required whatever method you decide for administration. 


e Decide upon the method of distribution. 


e By mail? Have you enclosed stamped self addressed enve- 
lopes to maximise the possibility of returns? 


e Telephone? Respondents are less patient sitting on the phone 
than in person. 


e Dropped off and picked up? 


e Will the questionnaires be anonymous? Do you need to check up on 
returns? Have you assigned code numbers? 
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e Maintain a master list of persons to whom the questionnaire is sent. 
Always place a contact number on the survey form. Respondents 
may have questions regarding the survey content or its administration. 


Ask yourself: 


. Given the issues being addressed in this program and consid- 
ering the costs of a survey, will a survey provide information 
which is important to this evaluation? 


Considering the characteristics of this community, is a survey 
of any sort practical? 


If so, should this be telephone, mail-out or distributed in 
some other way? 


Will the questions be open, closed or multiple choice? 


FOCUS GROUPS 


Focus groups provide a forum for open discussion of selected top- 


ics as a basis for an evaluation. 


THE PROS AND THE CONS OF FOCUS GROUPS 
The Pros: 


e Such discussion permits a dynamic exchange of ideas. 
Views expressed by one person can stimulate an opin- 
ion from another. 


e Structured and facilitated appropriately, the focus 
group can explore several issues in an in-depth way in 
a relatively short time. 
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e Although larger focus groups are not advisable, large 
numbers of people can be provided input through the 
use of an electronic touchpad. Each participant is 
provided with a numbered touchpad on which they 
enter the number which most accurately reflects their 
view on a particular structured question and answer. 
Feedback is immediate, the process is dynamic but 
tends to cost several hundred dollars for equipment 
rental and in remote communities additional travel 
costs. ; 


The Cons: 


e The groups must consist of a manageable number of 
people so a co-ordinator can facilitate the discussion. 
Six or ten and possibly 15 people may be suitable but 
two hundred is not. 


e The participants must be drawn from persons who have 
some interest or opinion on the issues Under discussion. 
lf the person is totally untouched or unconcerned 
about the issues then they are unlikely to have any 
opinion. 


e An assertive person may be able to dominate the dis- 
cussion. 


e The facilitator and the recorder must be adept at their 
tasks to ensure that discussion does not get ‘off-track’ 
and that all relevant information is recorded and ana- 
lysed. 


e Focus groups are not appropriate when working in Cul- 
tures, with persons or, perhaps, age groups which do 
not expect open, public expression of ideas and feel- 
ings. Elders may not want to share their views with oth- 
ers from the band or have their views challenged. 


e Focus groups are only effective when the topic is one 
with which are comfortable discussing in an open fo- 
rum. 


e Participants must be able to communicate in a com- 
mon language. 
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DEVELOPING AND RUNNING A FOCUS GROUP 


Developing.... 


e Determine a list of topics which are suitable for group discussion. 
There should be between three and seven. The number of issues ac- 
tually addressed depends upon the amount of discussion in the group 
given the time available. 


e Research the topic prior to the session. You will need to introduce the 
issue for discussion and often prompt and move the discussion on with 
examples of the matters being discussed. 


e Identify people who will be suitable participants. As we have seen 
above, the group should be a manageable number which will allow 
all to participate in the discussion. Those selected must also have an 
interest in the topics which are to be discussed. They may be partici- 
pants in the program, victims, members of the community, Elders, po- 
lice officers or program staff. The mix of person should reflect the vari- 
ous perspectives on the issues but allow for equal participation. 


e Locate a suitable space for holding the focus group. Participants 
should not have to worry about attending to work, family or band 
matters. Ensure that flip charts are available or that paper can be af- 
fixed to the wall. Provide felt tip markers. 


e Estimate the duration of the meeting. A half day might be appropri- 
ate, a full day may be acceptable but more than that and partici- 
pants will be reluctant to take time away from work and will be unable 
to concentrate. 


e Identify someone who is willing to take notes but who will not partici- 
pate. 


e Notify all participants and consider distributing background material 
prior to the meeting if this will facilitate discussion. 


and Running..... 


e Introduce the session, the objectives and the ‘rules of conduct’ (which 
might include everyone's right to be heard) and the requirements for 
confidentiality. Introduce the topics to be discussed and provide 
adequate background without introducing bias to the discussion. 


e Moderate the discussion by keeping the group on topic, encouraging 
participation, being supportive, stimulating discussion where appropri- 
ate but, remember, the objective is to hear the views of the partici- 
pants not to let others be exposed to your perspective. 


e Use flip Charts to summarise discussions. These can be used later as 
input to meeting notes. 
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e Some participants may not be able to read in the selected written 
language of the focus group. Assist these people by, for example, 
explaining the notes on the flip chart. 


e At the close of the focus group session ask the participants if there are 
any issues which have been missed. Thank the participants for their 
contribution. 


e Explain the next steps in the evaluation and what will become of the 
material which has been collected during the focus group session. 


Ask yourself: 


Given the nature of the issues addressed by this program 
would a focus group be an acceptable way to examine peoples 
views? 


Given the characteristics of this community including matters 
such as the culture, would a focus group provide an open ex- 
change of ideas? 


If so, what is the best make-up of the group to encourage an 
open exchange? 


Who do I have working on the evaluation who has the skills 
necessary to conduct a focus group? 


PERSONAL INTERVIEWS 


Interviews provide an opportunity for face to face questioning of a 
program participant, agency worker, community member or a 
representatives of other programs or agencies which may be im- 


pacted or have a view on the program being evaluated. 


There are two types of interview, structured and unstructured. 
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The structured interview requires the preparation of a series of ques- 
tions or specific issues to be Covered by the interviewer. The inter- 
view is focused and proceeds through the series of topics with the 


responses noted in detail. 


The unstructured interview allows more flexibility from both the in- 
terviewer and the interviewee. Although the general topics to be 
addressed are prepared ahead of time, the direction of the discus- 
sion is more free flow and cictated by the collective desire of the 
interviewer and the interviewee to examine the general issues 


which are the subject of the evaluation. 


THE PROS AND THE CONS OF INTERVIEWS 


The Pros: 
e Allow detailed descriptions of an event, feelings or per- 
spectives. 


e Face to face contact develops rapport and, possibly, 
more open communication. 


e Response rate to questions is far higher than alterna- 
tives as the potential respondent will find it hard to ref- 
use to participate in a one on one interview. 
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e The interviewer can use probing questions to explore at 
greater depth an issue raised by a respondent. 


e The body language of the respondent can be ob- 
served and the interview adjusted. 


The Cons: 


e Time consuming. Interviews may take between one 
and three hours. Writing up of notes is likely to take a 
considerable time. 


e = It is difficult to take detailed notes during an interview 
and most respondents will be reluctant to speak into a 
tape recorder. 


e The body language of the interviewer can be observed 
and the response adjusted. 


CONDUCTING AN INTERVIEW 


e Consider the people who work in, or are impacted in some way by, 
the program being evaluated. Decide how these people can pro- 
vide in-depth information which may be relevant to the evaluation. 


e Bearing in mind the objectives of the evaluation, prepare a list of issue 
areas on which the subject has special expertise or can provide a 
special perspective. 


e Turn each topic area into a question or series of questions. Develop 
follow-up questions which may be used as prompts. 


e Test the questions on a colleague who understands the need for con- 
fidentially. Amend the format to remove duplicate questions and to 
ensure clarity. 


e Make appointments with interviewees, briefly informing them of the 
context for the interview and provide an estimate of the time to be 
taken by the interview. If an interview takes more than one or two 
hours the respondent is likely to lose interest. 


e When meeting with the subject, describe the project and explain the 
confidential nature of the interview. Provide a brief overview of the 
topics to be covered and a time estimate. Keep to the time limitation. 


e Allow the respondent to speak freely guided by the interview topics 
which you have established. Use the prompt questions to delve 
deeper or to clarify something the interviewee has said. 


e Takenotes. Do not hold up the flow of the interview by laboriously 
writing everything which is said but make your notes sufficiently de- 
tailed that you can transcribe them later. Most people will be inhib- 
ited by the use of a tape recorder during the interview. 


e Donot debate the issues with the respondent. You are there fo listen 


to the contacts perspective on the topics. 
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e Remain aware of the respondents reaction to your questions. 


e Does he or she understand the question or the language you 
are using? 

e Does body language demonstrate that the person is uncom- 
fortable with the tone of the discussion? 


e At the close of the interview, thank the respondent for his/her partici- 
pation. Offer a contact number should there be a need to communi- 
cate with you. 


Ask yourself: 


Given the nature of the issues addressed by this program 
would personal interviews be an acceptable way to examine 
peoples views? 


Given the characteristics of this community including matters 
such as the culture and the personality of possible respon- 


dents, would a personal interview provide information valuable 
to the evaluation? 


If so, who should be interviewed to secure the information I 
need? 


Who do I have working on the evaluation who has the skills 
necessary to conduct interviews? 
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OBSERVATIONS- 


Observations are important to an evaluation. The evaluator is pro- 
vided the opportunity to view the program in operation. Observa- 


tion provides a practical context for the evaluation. 


THE PROS AND THE CONS OF OBSERVATIONAL WORK. 


The Pros: 


e The over-riding benefit of observations is that the 
evaluator is provided a practical perspective on the 
program which often cannot be achieved through 
other methods of information gathering. 


The Cons: 


e Inthe presence of an observer the subject, such as the 
agency worker or the program participant, is likely to 
act in a different way than when not being observed. 
This is particularly so when the evaluator becomes a 
participant observer and is involved, even in a periph- 
eral or passive way, in the program. The findings from 
the observation will, therefore, be a distortion of what 
normally occurs. 


e The subject of the observations is likely to ask question 
regarding the evaluation. It is difficult for the evaluator 
to respond without alerting the subject to the issues 
which are of importance to the evaluation. The sub- 
jects behaviour may change to address these specific 
matters. 


e There are occasionally questions of vicarious liability in- 
volved in accompanying some agency staff, particu- 
larly police, in their work routines. This may make ob- 
servational work with select groups difficult. 
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e Observational work is time Consuming. A level of trust 
must be developed between the observer and the ob- 
served before actions are ‘normalised’. Even when 
trust has been developed, the observations generally 
focus on the actions of one person or a small group of 
persons rather than the entire program. 


DOING OBSERVATIONAL WORK. 


e Carefully select the tasks to be observed. Observational work is time 
consuming and intrusive. 


e Make an appointment with the supervisor or manager to introduce 
the objectives of the evaluation and the reason for the request to ob- 
serve. 


e Be prepared to sign a waiver absolving the agency of any liability in 
the case of injury. 


e Inform the worker that you are not assessing his/her performance but 
trying to understand how the program or process works. Ensure the 
worker that any observations on activities will not be attributable to 
the individual worker. 


e Observe only. Avoid participating as it is the worker's actions that you 
are there to observe. Avoid making comments which may suggest 
what the worker should be doing or any comment which may an- 
tagonise the offender/victim/witness or citizen. 


e Donot actively take notes. This will be very off-putting to the worker. 
Try to remember what is said and done and make notes later. 


e Ask approval before viewing files or entering sensitive areas. You are 
a visitor in the workers space and routine. 


CONSIDERATIONS IN CHOOSING FORM OF EVALUATION 
As we have seen, there are a variety of methods or research tools 


to assist you in gathering information. It is unlikely, except in major 


evaluation projects, that you will use all types of data gathering. 
So, in deciding which approach to take, consider the following. 


e What are the skills of those who are working on the evaluation? Some 
people may be quite comfortable in front of a group running a focus 
group. Others may be more comfortable with one on one interviews. 


e What are the skills and preferences of the subjects? Some people 
may be unwilling to give an opinion on issues in a public way, others 
may be very nervous with a one on one interview. 
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e What information already exists as a normal part of operations? If 
data gathering is necessary, what inconvenience and resistance 
would this receive from the workers? 


e What funding do you have? Some approaches cost more money 
than others. 


e What is the cultural context of the evaluation. Do the majority of citi- 
zens have telephones? Is everybody in the community comfortable 
communicating in the same way and use the same dialects and can 
equally participate in a survey? 


Where possible, use more than one information source for the data 
which is used in the evaluation. This will helo to demonstrate that 


the conclusion of the evaluation report are well founded. 


Ask yourself: 


Given the types of issues addressed by this program and the 
specific nature of this method of information gathering , will 
observational work contribute information valuable to the 
evaluation? 


If so, which places in the process of agencies dealing with 
these issues would be most appropriate to undertake observa- 
tional work? 


Do I have the skills to do observational work? 


Will I be able to get approval to do the work? 
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We have now reviewed the types of information which you may 
find useful in undertaking your evaluation. It is unlikely that you will 
use all these types. Only in the most comprehensive (and well 
funded) evaluation would you employ all these types of informa- 
tion gathering tools. You must consider the size of the program you 
are evaluating and the issues which it is designed to address to de- 
termine which approach or approaches is required to obtain the 


information which you need. 


An important point to remember is to consider the culture and 
characteristics of the community and the preferences of those who 
may be asked to provide information. The information gathering 
tools must be tailored to these needs. Some Communities may not 
have access to telephones, may use primarily oral forms of com- 
munication and some people may be reluctant to share their views 


in public. 
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Ask yourself: 


What skills do I have or the people working with me and what 
method of information gathering will work best? 


What information was used in other communities where a simi- 
lar evaluation was undertaken? 


What types information will best address the objectives which 
were set for the program and which I have set for the 
evaluation? 


Considering the indicators that I have decided are ideal to 
evaluate the program, where can I find that information? 


Does the information already exist? 


If that specific information is not readily available, is it fea- 
sible to develop the methods to gather the information, do I 
have the skills or the money to successfully complete this 
task? 


Is it reasonable to expect agency workers to collect this in- 
formation in addition to the other data that they collect as a 
routine function of their job? 


Given the particular nature of this community what types of 
information gathering will yield best results? 


If it is not feasible to collect the information, how can I 
change the evaluation plan so that the end result of the 
evaluation is still valid? 


You have taken a lot of time and effort to get an understanding of 
what are your options for conducting your evaluation. You now 
have sufficient information regarding approaches to assess where 
you are in the evaluation planning. 
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Planning Your Work and Working Your Plan. 
At this point in your planning: 


e = |Ithas been decided that there is a community concern which must 
be addressed. 


e A program has been developed and implemented. 


e Either the program planning accommodated an evaluation or it has 
been necessary to review the program and begin the evaluation 
planning after the introduction of the program. 


e You have reviewed the original proposal and program action plan. 


e You have conducted a search of information sources such as libraries, 
internet, contacts in local agencies to obtain information concerning 
the issues with the program is designed to address. 


e You have reviewed available literature regarding the operation of 
similar programs in other communities. 


e You have reviewed information regarding the evaluation of similar 
programs. 


e You have spoken with the major stakeholders to gain an understand- 
ing of how the particular program was intended to operate, and, 


e You have reviewed the methodologies for undertaking evaluations 
and understand the options, benefits and disadvantages of the vari- 
ous methods. 


You are now ready to develop a evaluation plan...............606 
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PREPARING A WORK PLAN FOR THE EVALUATION AND UNDERTAKING THE WORK 


Having gathered the relevant material regarding your options you, 
or if you are not working alone, the co-ordinating group, should 


meet to develop a specific action plan. 


The plan template on the following page will provide some guid- 


ance for structuring the evaluation approach. 


This is also a logical place to consider the budget which you require 
to undertake the evaluation. The plan will either assist you in pre- 
paring a proposal or help you in deciding which aspects of the 
plan will have to be sacrificed to meet budget restrictions. If the 
assigned budget is low, you will have to decide whether the ab- 
sence of funding and the corresponding reduction in potential 
sources for information will impact the validity of the evaluation to 


the point that continuation is futile. 
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Evaluation Plan: 
e Program title: 
° Brief description of the program based upon; 
e What( problem is the program intended to address) ¢ 
e What (is the objective of the program) 2 
e What (are the indicators) ? 
e Who (is the program directed at, the subjects) ¢ 
e Who (is expected to benefit) 2 
e When (is the program intended to bare results) ? 
e Where (is the program supposed to have impact) 2 
e How (is the program intended to have impact) ? 
e What is the form of the evaluation? 
° (Needs Assessment) 
e Process evaluation 
e Impact evaluation 
e What type(s) of data should ideally be used? 


Quolita- Quantitative Detail specific Currently available - 
tive information required where? 


Detail specific Prospective source 
information required for information.. 


e Whois responsible for finalising the development of the data gathering fools? 

e What is the time schedule for finalising the data gathering? 

e Who is responsible for each type of data gathering and analysis? 

e Whois responsible for preparing the draft and final reports? 

e What is the schedule for undertaking the evaluation and has a project tracking sheet 


been developed 2 
e Who will be receiving copies of the report? 


e@ Whois the principal contact for inquiries regarding the evaluation? 
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The plan should become the overall guide for the evaluation. The 
plan can become the front sheet for the co-ordinators file in which 


the many components of the evaluation will be collected. 


Each of the tasks should be assigned to a member of the co- 
ordinating group unless the responsibility is to be assumed by the 


sole evaluator. 


Where, for example, a survey must be developed or a data gath- 
ering form has to be designed for statistics, this task should be as- 


signed to a lead person who may engage others fo assist. 


Project tracking 


In complex, multi-task evaluations it may assist the co-ordination if 
the work plan is put onto a schedule plan. There are several so- 
phisticated project tracking tools available such as, Gannt or PERT 
charts. Many of these project tracking tools are available for 
computers but most basic evaluation plans can be managed 


manually. 


The important things to track are the tasks, with the person respon- 
sible (on the left side of a chart) and the time scale (along the top 
of the chart). Beside each task draw a line under the appropriate 


time period to show the start, duration and end of the task. 
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A Word On The Personal Approach 


The success of the evaluation Is partly dependent upon the per- 
ception of the evaluating staff by the subjects of the evaluation 
and other stakeholders. Those undertaking the work must not only 
have a workable, valid plan, they must approach their work in such 
a away that they encourage the trust and support of those who 


are asked to participate in the evaluation. 


An ethical approach to the work is important. All resoondents must 
be aware of the nature of the evaluation in which they are to be- 
come involved. They must also understand for what purpose the 


information is being gathered. 


The confidentiality of information is a foundation for the evaluation. 


This is particularly important in small communities. 


Some cultures respond in different ways to various approaches. 
Some approaches which require open, uninhibited discussion may 
not be suitable for some participants. People respond differently to 
gestures and body language depending upon their cultural back- 
ground. There are fundamental differences between aboriginal 
and non-aboriginal cultures and between aboriginal cultures. Be 


respectful of people’s heritage. 


If you are a visitor to an organisation for the purposes of Undertak- 
ing the evaluation, be as unobtrusive as possible and do not disrupt 


routine. 


Perivale + taylor 


First Nations Crime Prevention — Self Evaluation Manual 67 


RULES TO MAINTAIN INTEGRITY AND CONFIDENTIALITY 

It may be necessary for the co-ordinating group and even for the 
individual evaluator to develop terms of reference which establish 
guidelines for the conduct of the work. 

e All participants are informed of the confidential nature of the work. 


e Records, notes from interviews or other documentation are kept in a 
safe and secure location. 


e The confidentiality of any information gathered is honoured and is 
seen by everyone involved to be honoured. 


e The approach of the evaluator must be seen to be objective, com- 
ments concerning the program should not reveal opinion which may 
appear to bias the findings. 


e Negotiation of access to data or arrangements regarding interviews 
or observations of job tasks must be carried out in good faith. Com- 
mitments and promises must be honoured. 


e = Any information which is included in any interim or final report is re- 
vealed in aggregated form, that is the response of any one individual 
is not revealed. 


The evaluator or the co-ordinating group must closely monitor the 
evaluation process. Adjustments may have to be considered if 
one aspect of the evaluation fails fo achieve the success that had 
been hoped. Rigidity in the work plan may impede the accom- 


plisnment of the objective of the evaluation. 


The co-ordinator or co-ordinating group play an important role in 
ensuring that the work is progressing as planned. Periodically, 
meetings should be held to, 


e discuss the progress of data gathering and whether any adjustments 
have to be made to the schedule or methods, 


e  toascertain that all prospective subjects or contacts for the evalua- 
tion are assisting as hoped or whether facilitation is required, and, 


e discuss the data gathered to determine potential interim findings. 
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Ask yourself: 
Have I prepared a work plan, and decided 
which types of information I will need, 
where I can get that information, 


who is responsible for developing the information gathering 
tools, 


who will conduct the work, and 
with people and time lines scheduled? 


Informed everyone involved of the requirements for confiden- 
tiality and cultural sensitivity? 


Begun gathering the information which will be the basis for 
the evaluation? 


Making Sense Of The Information Gathered 


By now you will have collected piles of information from your differ- 
ent sources. You may have interview notes, and numbers drawn 
from police records, notes from focus groups and, perhaps, even 


survey results. Confusing and overwhelming! 


You must now find meaning in the data which you have gathered. 
Depending upon the size of the evaluation project you are likely to 
have collected large amounts of information. Summaries which 
describe aspects of the program, charts fo illustrate processes or 


statistics will help in the analysis of the information gathered. 
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Keeping in mind the objectives of the evaluation, collect and con- 
sider the groups of information which you have collected. Some- 
times even writing the evaluation objective(s) on a large piece of 
paper on the wall will assist the process of focusing on the objec- 


tive while sifting through the material. 


The information gathered may have to reorganised “out of" the 
form in which it was collected. Material gathered in interviews will 
be combined with information collected from files, statistics may 


tie-in to the results of the survey or your observations. 


To assist this analytical process it may help to see the complete 


program in terms of input, process and outputs. 


Inputs - Refers to the participants in the program, the persons or is- 
sues which are meant to be addressed, the skills, equipment and 


structures which are providing support for the program. 


Process - Refers, as the name implies, to the activities which occur 
in the process of operating the program. This information relates to 
the delivery of the program and considered factors such as the 
number of clients, the number of calls, how files are maintained, 
the treatment of victims by agency workers and other activities 


which move the program through its intended course. 


Outputs - Outputs focuses upon the impact of the program and 
should include issues such as whether the number of target inci- 
dents declined, whether the participants are satisfied with the 
treatment, whether the staff were trained and, at a general level, 


whether the community ‘feels’ safer. 


Perivale + taylor 


First Nations Crime Prevention — Self Evaluation Manual 70 


GETTING STARTED 


e Those people co-ordinating the evaluation should meet. The evalua- 
tion will benefit from a wide range of views of those involved in data 
collection. The evaluator working alone may still benefit from discuss- 
ing the information with another person. This confidante should not be 
a subject of the evaluation and must understand the requirements for 
confidentiality. 


e Set aside preconceived ideas about the success or failure of the pro- 
gram being evaluated. 


e Arrange the information in groups of data such as the input, process 
and outputs suggested above. 


e Clarify the information, assess its relevance to the objective of the 
evaluation. 


e Review the information available and ask whether there is sufficient 
data to answer the evaluation questions. 


e Look for clusters of information or patterns, trends or themes in the 
data. 


e Look for statistical information which may indicate changes in the fre- 
quency of occurrences as a result of the crime prevention program. 


e Use a flip chart to note themes and list relevant information and infor- 
mation sources under each theme. 
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e See if there are any conclusions which can be drawn from the infor- 
mation. Start with very basic questions such as 


e Has this program worked?2, 

e Why or why note 

e What pieces of information lead to this conclusion2 

e Does any data conflict with this or these conclusion(s) 2 


e Is/are the conclusion(s) correct or is there some other or addi- 
tional conclusion which can be drawn2 


e List the reasons for your conclusions and follow the thread back 
through your data and the outputs, process and inputs to understand 
the various reasons for the success or failure of the program. Look for 
a logical chain of evidence to account for your belief. 


e = If the information is the basis for a major conclusion, establish that it is 
reliable and supported by other data possibly from another informa- 
tion source. 


e Again, review the information available and ask whether there is suf- 
ficient data to answer the evaluation questions. 


e Consider whether it is necessary to return to the data gathering por- 
tion of the evaluation and seek clarification of any issues, do you need 
more interviews? 


e Ask whether there are there any other reasons for the conclusion 
which you have drawn. Keep your mind and options open! Even 
consider possibilities outside the original breadth of the evaluation. 


e Determine whether there is any inconsistency or any bits of information 
which contradict your conclusions and should the conclusion be 
changed, explain this apparent contradiction. 


e Review the information which you have not actively used in drawing 
you conclusions. Determine whether there are any additional conclu- 
sions which could be drawn from this material or whether the informa- 
tion supports any other conclusion. 


e List your conclusion and discuss ways to address these in the form of 
actions to be taken. These will be your recommendations. The con- 
clusions are only a starting point for action. 


e Summarise the findings with clear explanations for the conclusions. 
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Ask yourself: 


Have I met with my co-ordinating group and sorted the infor- 
mation to themes or issues and listed the particular informa- 
tion or evidence which supports these findings? 


Am I satisfied that my conclusions are supported by the data? 


Depending upon the particular information gathering tools that 
I used, is each conclusion supported by various sources such as 
interviews, surveys and file review? 


If there are inconsistencies, can I explain why? 
Are my conclusions clear and precise and logical? 


Have I addressed the evaluation objective and focused on the 
indicators? 


Are my recommendations do-able and is responsibility as- 
signed? 


Preparing The Evaluation Report 


The report informs the stakeholders of the results of the evaluation. 
The report is important because it will be Used as a basis for major 
decisions regarding the effectiveness of the staff involved, the suc- 
cess of the program in meeting the original objectives, the adjust- 
ments which may be required to refocus the program and even 
continuation of the program. 


USE OF THE DRAFT REPORT 


A draft report should be prepared for discussion with major 
stakeholders. This serves several purposes, 


e It allows the evaluator to collect his or her thoughts and to organise 
them in a logical fashion prior to the final version. 
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e It permits discussion of the findings with the major stakeholders so that 
they can offer an opinion on the principal findings. The stakeholders 
may provide a different perspective on the findings which may point 
to a failing in the research approach. 


e Discussing the draft with the major stakeholders may permit the inclu- 
sion of an appendix in the report which outlines remedial steps which 
have been taken to address matters raised in the evaluation. 


THE REPORT SHOULD CONTAIN: 


Summary - This will provide a brief program description, synopsis of 
the objective of the evaluation, the principal findings and the 
sources upon which the consultations are based. The summary or 
executive summary provides an easily accessible description for 
those who have neither the time nor the need to review the full re- 
port. 


Introduction- Describes the goals of the program, the purpose of 
the evaluation, the duration of the work and the staff involved. 


Methodology - Describes the process for conducting the evalua- 
tion including your sources of information. This will not refer to the 
people who supplied information but the general information 
gathering approaches which you used. 


Principal findings - An evaluation report must include description 
and judgement. Describe what was found in the evaluation, what 
conclusions were drawn from the study and the reason for these 
conclusions. 


Recommenaations - Provide suggestions for ways in which the pro- 
gram can be changed to make it more effective. This should also 
be a proposal for actions which should be taken to address the 
original community crime issues. The recommendations make the 
report a living document which creates expectations for action by 
those responsible for, or involved in, the program. 


Prioritise the recommendations and suggest the action required to 
resolve the issue and the person or position responsible for the ac- 
tions. Remember, the report is not to highlight blame but to pro- 
vide direction to ensure that the community crime problem is re- 
solved. 
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Conclusion - The summary rounds off the evaluation project by 
briefly connecting the main themes of the evaluation and the re- 
sulting report and tying these to the conclusions. 


Bibliography - Credit should be given where the evaluator has re- 
ferred to previously published material. The bibliography also 
demonstrates to the reader the breadth of background work un- 
dertaken in the course of the evaluation. Style manuals are avail- 
able to provide the format for listing reference works, however, the 
most important thing is that documents, including files and Unpub- 
lished reports are listed. 


Where confidentiality is not an issue, consider recognising those 
who contributed to the evaluation by providing information or in- 
put. 


Final Report And Feedback 


Following the release of the final report, the evaluator and the 
stakeholders or agency workers may wish to meet and discuss the 
findings. The feedback session will allow staff who have been in- 
volved or certainly are acquainted with the conduct of the evalua- 
tion to receive feedback from the evaluator. 


ELEMENTS OF FEEDBACK 


The feedback session should: 


e Be based upon an informed understanding of the contents of the re- 
port. Consider distributing a synopsis of the findings prior to the meet- 


ing. 
e Be structured rather than a free for all. 


e Be facilitated to prevent the session becoming an assault on the 
evaluator if the report is critical 


e Not assign blame to anyone who may have inhibited the effective- 
ness of the program 


e Beconstructive and positive by explaining weakness but also offer so- 
lutions through recommendations. 


e Allow staff or stakeholders to respond to the findings of the report by 
focusing on actions for improvement rather justifying past behaviours 
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Distribution of the Report 


Recognising the degree of confidentiality and sensitivity of the 
Evaluation Report, there are several prospective recipients of 
copies of the report findings. 


Who receives a copy is dependent upon the amount of local in- 
terest in the program being evaluated. This interest may be 
broadened by the nature of issues being addressed in the findings. 
For example, if the evaluation highlights controversial issues such as 
‘staff inadequacies’ then the findings become more sensitive and 
public and media interest in the report may increase. Be prepared 
to support your conclusions in a public forum. 


An evaluation of a pilot crime prevention program may generate 
significant interest outside the local area and may result in requests 
for copies of the final report. 


WHO SHOULD GET FINAL REPORT 
e The Program Manager for Program Staff. 
e The funding agency. 


e The Governance authority such as the Board of Directors or the Police 
Services Board. 


e The Band Council, the municipal, provincial or federal agency with 
prime responsibility for the action of the agency. 


e The local media. 
e Local libraries or academic institutions. 


e Presentation at a conference so that other communities may benefit 
from the experience of your community. 


e Publication in a magazine or journal or discussion of the findings on 
local radio. 


However, remember the more copies of a report that are required 
the greater the cost. You may consider only distributing an execu- 
tive summary to certain people. Permission may be required from 
the holder of the copyright prior to distribution of the final report of 
the evaluation. 
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Ask yourself: 


Have I prepared a draft report which is based upon the find- 
ings of the evaluation and the discussions at the group infor- 
mation analysis and structured the report according to the 
suggested format? 


Have I discussed the draft report with major stakeholders 
and considered their feedback? 


Have I decided upon the distribution list for the final report 
and made sufficient copies ? 


Follow-up 


The preparation and presentation of the final report does not nec- 
essarily mark the end of the project. Follow-up is required in sev- 
eral areas. 


Follow through of recommendations - lf responsibility for recom- 
mendations has been assigned, it may be the evaluator’s task to 
change roles and ensure that recommendations are addressed. 


Meala contacts - The media may request access to the evaluator 
to discuss the findings. The media can also be incorporated in 
generating interest and action as a result of the report. You should 
consider an item in the community or Band Council newsletter. 


Thanks - Depending upon the breadth of the evaluation, many 
people may have participated in the process by supplying data, 
allowing the evaluator to observe their work routines, participating 
in focus groups or providing insight fo the evaluation process. Al- 
though it may be impossible to contact everyone who participated 
in the evaluation project, a simple note of thanks with a mention of 
the outcome of the work may leave participants with a positive 
disposition towards participating in future evaluation projects. A 
community feast or social may be appropriate if a large portion of 
the community has contributed to an evaluation. 


Self-evaluation - For many, the co-ordination of an evaluation proj- 
ect is anew experience. The evaluator is required to learn different 
skills and to become acquainted with terms and practices which 
are new. The evaluation project is often stressful and time con- 
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suming. Since the continuation of a program, along with jobs, is 
often dependent upon the findings of an evaluation the task is 
also one which is associated with significant responsibility. 


The process of program development, program operation evalua- 
tion and feedback is really a circular process. The recommendoa- 
tions will feed into agency or community decisions regarding which 
crime problems to tackle and how to develop a program. 


At the end of the project, you should allow time to reflect upon the 
process. 


Ask yourself: 
Did I conduct this evaluation in a fair and objective way? 
Did I treat all the participants fairly and equally? 


How can I improve from this experience when I undertake the 
next evaluation? 


How can my work on this project reap the greatest benefit to 
my community and to crime prevention? 


Now take action to improve the program and your com- 
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Appendices 


BUDGET 


Consider: 


e Salaries. \f not volunteer or regular agency staff, researchers 
salaries, honorarium, consultants fees, clerical or administrative 
salaries. 

e Accommodation. Rental costs for office or focus groups 

e Supplies: Office materials including, paper, printer cartridges, 
writing tools, flio chart paper, file space, storage 

e Office equioment. Fax, computer, telephone, 

e Other equioment rentat computer, touchpad technology for 
focus groups. 

e Operational telephone particularly long distance, e-mail, Inter- 
net access fees, postage/courier for documents from/to other 
communities, computer time for data analysis. 

e Jravet Visits to other communities, travel around your commu- 
nity, travel by others such as peer evaluators or consultants. 

e Printing. Surveys, correspondence, draft and final reports. 
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EVALUATION CHECKLIST 


Tasks to do Person 
responsible 
. Organise co-ordinating committee (or evaluator working 
alone 


1 
2. Was an evaluation part of the original program plan? 
What was planned? 


3. Define the program Objectives. 
4. Define the Evaluation Objectives (same?). 
Sy, 


Establish the type of evaluation you need? 
° Process? to find out about the operation of the 
program. 

e Impact? To understand the final effects of the pro- 

@ People 

° Funding 
8. Determine possible sources and apply. ieee aa 
9. Has the project plan form been prepared? 


10. Identify the most appropriate types of information gath- 
ering, taking account of the program issues, personality of 
those involved and the culture of the community. 

e Statistics 
e File Review 
e Survey 


Scheduled 
finish date 


Done 
of 


gram upon the community problem 
° (Remember, a Needs Assessment is usually some- 
thing that is done before a program is developed 
to determine what type of program is required 
Determine who should do the Evaluation. 
° Evaluator only 
° Program manager 
e Program staff 
® 
e 


Community member 
External consultant 

Peer from another communit 
7. Determine whether other resources needed. 


Focus Group 


Personal Interview 


Observations 
11. Detail information sources, the agency and contact per- 
son + check if approval is required 


12. Assign responsibility for each inf. gathering task taking into 
account the skills of the people who are working on the 
evaluation. 


13. Assign timelines for each stage of the evaluation and 
enter on the project schedule. 
14. Undertake research, gather all information. 


15. Meet to analyse all information. 


16. List conclusions with all the relevant support information. 


17. Prepare draft report and discuss with the principal 
stakeholders. 
18. Prepare final report with recommendations. 
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GLOSSARY OF TERMS 
Analyse -To examine information to show its essence or meaning. 


Baseline data - Information that is already known. A point from 
which to measure, a standard. The state of affairs before the pro- 
gram began. 


Before-and-After Comparison Measures variables like ‘crime 
rate’ or ‘levels of fear of crime’ before the program implementa- 
tion and compares those figures to the figures collected once the 
program has been established or completed at any given point. 
Also called ‘pre-and post-’ comparison. 


Closed Questions -Questions which are answered with a ‘yes’ or 
‘no' or some other question with a restricted range of options as 
Answers. 


Culminating data - The information (statistics or other forms of data) 
which are collected at the termination of a program and, com- 
pared with baseline data, used as a basis for determining the im- 
pact. 


External Evaluation - An evaluation conducted by someone who is 
not considered a community member or program participant, Usu- 
ally considered an outside expert evaluator. 


Evaluation - To examine and judge based upon performance, re- 
sults and /or impacts. Evaluation allows you to find out how valu- 
able something is but more so it provides you with meaning. It's 
empowering in that it equips you with information that can allow 
you to make sound judgements and decisions. Evaluation is an 
important tool for managing service delivery. 


Evaluation Approach - Outlines how you plan to carry out an 
evaluation. 


Feedback - Information that is provided back to the group of 
people such as agency workers whose agency has been evalu- 
ated. Feedback provides information regarding the assessment of 
what is going on and what they have been doing. 


Focus Group - A small group meeting intended to facilitate the dis- 
cussion of specific points relevant to the evaluation by people with 
an active interest in the evaluation issues. 
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Formative Evaluation -Used to make improvements or changes to a 
program while it is in progress. Questions asked are: What is work- 
ing? What needs to be improved? How can it be improved? 


Needs Assessment - The preliminary stage of setting up a program 
to develop and understanding of the breadth or structure and form 
of a required program based upon such tools as an environmental 
review, and problem analysis. 


Impact Evaluation - The assessment of the effect of a program, the 
degree to which the program has changed the circumstances 
which is was set out to address. The Impact evaluation may also 
go broader into other unanticipated effects. 


Indicator - A suggestion, like an event or statistic, used to measure 
the success of a program in meeting its objective. 


Literature Search - The review newspaper articles, periodicals, 
academic journals, reports secured from other agencies and Inter- 
net materials relevant to the crime prevention program operation 
or evaluation to secure background information in preparation for 
the development of your plan. 


Objective - An objective is a goal you set which you try fo 
achieve. It is a statement of what you want your program to do. 


Open Questions - Those questions that can be answered with de- 
scription and opinion. 


Planned vs. Actual Performance - A project is evaluated against its 
objectives. Therefore, it compares what it has set out to do with 
what it actually has done. 


Process data- The information which is collected over the duration 
of a program either as a normal part of business or as a conse- 
quence of a data gathering tool for the purposes of the evalua- 
tion. 


Process Evaluation - Involves self-examination or assessment of the 
organisation’s structure, how it runs, who is involved, board struc- 
ture and operational issues. 


Qualitative Data Basic attributes or features of a program usu- 


ally expressed by opinion regarding the quality of the program. 
Looking at the meaning and structure of a program ("soft "data). 
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Quantitative Data The measurement of a program in tangible 
term, using numbers/statistics ("hard" data). 


Self-evaluation - Evaluation conducted by individuals working in 
the program; a participant in the program; people who belong in 
community but not necessarily working in the program; a 
stakeholder in the program. 


Summative Evaluation - An assessment used to judge the usefulness 
or worth of a program and is conducted near the end of a pro- 
gram. This can be done at a community level where a single proj- 
ect is evaluated or it can be done at a regional or national level 
whereby the impact of all projects are compared such as the Abo- 
riginal Head Start evaluation. 


Terms of Reference - The Rules of Conduct, the evaluation parame- 
ters, aroad map for the evaluation that will be conducted which 
outline the obligations of all participants in the evaluation. They 
should reflect evaluation objectives and priorities and should indi- 
cate the kinds of general questions the evaluation should cover 
such as a evaluation objectives, time frame for evaluation, re- 
sources to be committed, list of deliverables. 
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Questions To Guide The Evaluation Process 


How To Use The Manual. 
What is the purpose in undertaking this evaluation? 


Given the progress of the program, is it the appropriate time 
to undertake an evaluation? 


Do I have the resources as far as skills and funding? 


If I am doing the evaluation alone, can I organise an advisory 
committee from the community? 


Getting Ready. 


Undertaking the Background Research 


Have I fully described this program as it is intended to oper- 
ate in this community? 


Have I defined in a concise way the objective(s) or goal(s) of 
the program? 


Have I described the important community factors which may 
influence the operation of this program? 


Do I understand if, and how, the program operates in other 
communities? 


Setting Evaluation Objectives: 


Have I described the objectives of the program or, if these 
were not specified in the original program outline, have I de- 
veloped objectives for the evaluation? 


Have I determined what types of information will be used to 
indicate whether, and how, the program has worked? 
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Deciding the Form of the Evaluation Program 


Given the nature of the program and the stage at which I am 
planning the evaluation, which approach or approaches will best 
allow for an examination of the effects of the program? 


Deciding the Type of Information Upon Which Your Evaluation 
Will Be Based 


Considering the nature of this crime prevention program and 
the sorts of information which might be available, which ap- 
proach or combination of approaches will provide the best 
foundation for an evaluation? 


What sorts of information are likely to be available in this 
community? 


Given the type of people in my community and/or the staff 
working in the program, how am I likely to get the most valu- 
able information? 


Where Do We Find The Information That We Need? 
STATISTICS RESEARCH 


Given the objectives of this program what sort of stats infor- 
mation do I need? 


Do any of the community agencies, such as police or the Band 
Council, compile stats. which may be useful in this evaluation? 


If the stats which are kept are not entirely suitable, is it pos- 
sible to develop a process to collect stats. information for the 
purpose of this evaluation? 


FILE REVIEWS 


Given the objectives of this program what sort of file infor- 
mation do I need? 


Do any of the community agencies, such as police or the Band 
Council, keep file information which may be useful in this 
evaluation? 


If the files which are kept are not entirely suitable, are the 
required files available in any other location? 
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Do I need any sort of security clearance to have access to this 
material? 


SURVEYS 


Given the issues being addressed in this program and consid- 
ering the costs of a survey, will a survey provide information 
which is important to this evaluation? 


Considering the characteristics of this community, is a survey 
of any sort practical? 


If so, should this be telephone, mail-out or distributed in some 
other way? 


Will the questions be open, closed or multiple choice? 
FOCUS GROUPS 


Given the nature of the issues addressed by this program 
would a focus group be an acceptable way to examine peoples 
views? 


Given the characteristics of this community including matters 
such as the culture, would a focus group provide an open ex- 
change of ideas? 


If so, what is the best make-up of the group to encourage an 
open exchange? 


Who do I have working on the evaluation who has the skills 
necessary to conduct a focus group? 


PERSONAL INTERVIEWS 


Given the nature of the issues addressed by this program 
would personal interviews be an acceptable way to examine 
peoples views? 


Given the characteristics of this community including matters 
such as the culture and the personality of possible respon- 
dents, would a personal interview provide information valuable 
to the evaluation? 


If so, who should be interviewed to secure the information I 
need? 


Who do I have working on the evaluation who has the skills 
necessary to conduct interviews? 
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OBSERVATIONS- 


Given the types of issues addressed by this program and the 
specific nature of this method of information gathering , will 
observational work contribute information valuable to the 
evaluation? 


If so, which places in the process of agencies dealing with 
these issues would be most appropriate to undertake observa- 
tional work? 


Do I have the skills to do observational work? 
Will I be able to get approval to do the work? 


What skills do I have or the people working with me and what 
method of information gathering will work best? 


What information was used in other communities where a 
similar evaluation was undertaken? 


What types information will best address the objectives which 
were set for the program and which I have set for the evalua- 
tion? 


Considering the indicators that I have decided are ideal to 
evaluate the program, where can I find that information? 


Does the information already exist? 


If that specific information is not readily available, is it feasi- 
ble to develop the methods to gather the information, do I 
have the skills or the money to successfully complete this 
task? 


Is it reasonable to expect agency workers to collect this in- 
formation in addition to the other data that they collect asa 
routine function of their job? 


Given the particular nature of this community what types of 
information gathering will yield best results? 


If it is not feasible to collect the information, how can I 
change the evaluation plan so that the end result of the 
evaluation is still valid? 
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Planning Your Work and Working Your Plan. 
Have I prepared a work plan, and decided 
which types of information I will need, 
where I can get that information, 


who is responsible for developing the information gathering 
tools, 


who will conduct the work, and 
pe oe with people and time lines scheduled? 


Informed everyone involved of the requirements for confiden- 
tiality and cultural sensitivity? 


Begun gathering the information which will be the basis for 
the evaluation? 


Making Sense Of The Information Gathered 


Have I met with my co-ordinating group and sorted the infor- 
mation to themes or issues and listed the particular informa- 
tion or evidence which supports these findings? 


Am I satisfied that my conclusions are supported by the data? 


Depending upon the particular information gathering tools that 
I used, is each conclusion supported by various sources such as 
interviews, surveys and file review? 


If there are inconsistencies, can I explain why? 
Are my conclusions clear and precise and logical? 


Have I addressed the evaluation objective and focused on the 
indicators? 


Are my recommendations do-able and is responsibility as- 
signed? 


Preparing The Evaluation Report 


Have I prepared a draft report which is based upon the find- 
ings of the evaluation and the discussions at the group infor- 
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mation analysis and structured the report according to the 
suggested format? 


Have I discussed the draft report with major stakeholders 
and considered their feedback? 


Have I decided upon the distribution list for the final report 
and made sufficient copies ? 


Follow-up 
Did I conduct this evaluation in a fair and objective way? 
Did I treat all the participants fairly and equally? 


How can I improve from this experience when I undertake the 
next evaluation? 


How can my work on this project reap the greatest benefit to 
my community and to crime prevention? 


Se Set Now take action to improve the program and your com- 
munity! 
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